Image processing device, image processing system, image processing method, and program

ABSTRACT

An image processing device, an image processing system, an image processing method, and a program capable of suppressing unnaturalness of color of an object extracted from a color image are provided. An image processing device includes an object area determination unit that determines an object area, a mask image generation unit that generates a mask image, an object image generation unit that generates an object image on the basis of the object area, a color determination unit that determines a color to be subtracted from colors to be applied to the object area and sets a smaller number of colors than those of the object area as colors of the object image, and a probability calculation unit that calculates a probability of pixels of the mask image being pixels of the object area, and the object image generation unit sets the colors set using the color determination unit as colors of an edge area of the object image on the basis of the probability.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority under 35 U.S.C § 119 to JapanesePatent Application No. 2018-178970 filed on Sep. 25, 2018, which ishereby expressly incorporated by reference, in its entirety, into thepresent application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image processing device, an imageprocessing system, an image processing method, and a program, and moreparticularly to area extraction of a color image.

2. Description of the Related Art

There is known a scheme of extracting an object from a color image suchas a photograph. The object extracted from color images can be used forcreation of a stamp or the like.

“Line Creators Studio”, Internet <URL:https://creator.line.me/en/studio/> describes a scheme of capturing animage captured using a smartphone from an album, designating a trimmingrange, extracting an object, and creating a stamp. However, in thisscheme, it is necessary to directly designate a contour of an object,which takes time and effort. On the other hand, there is known a schemeof extracting a contour of an object from an image without directlydesignating the contour of the object using graph cutting and deeplearning.

JP2014-071666A describes an image processing device that calculates an αvalue representing a degree of transparency on the basis of aprobability of a pixel belonging to a foreground image, and multipliesthe α value by a pixel value to extract a foreground image. Further,JP2017-220098A discloses generating a contour candidate probabilityfield in which an infinitely small cost is given to pixels adjacent toeach other on a contour line of a foreground area, and weighting aninitial random field in the contour candidate probability field togenerate a process probability field.

The image process device described in JP2017-220098A sets the processingprobability field as a cost function, obtains a combination in which acost in a case where a foreground or a background is labeled for allpixels of an original image is minimized, using graph cutting, separatesthe combination into a foreground and a background, and extracts theforeground area.

SUMMARY OF THE INVENTION

However, pixels belonging to an object extracted from an image andpixels belonging to a background are both present in a boundary betweenthe object and the background. Then, color of the background or the likeis mixed into the object at the boundary between the object and thebackground, and the color of the object becomes unnatural. Further, inan object extracted from a high definition color image, unnaturalness ofcolors of the object is remarkable.

JP2014-071666A, JP2017-220098A, and “Line Creators Studio”, Internet<URL: https://creator.line.me/en/studio/> do not disclose a problem thatthe color of the object becomes unnatural and do not disclose atechnology for solving the problem. That is, an object or the likeextracted by applying the technology described in JP2014-071666A,JP2017-220098A, and “Line Creators Studio”, Internet <URL:https://creator.line.me/en/studio/> may cause color unnaturalness.

The present invention has been made in view of such problems, and anobject of the present invention is to provide an image processingdevice, an image processing system, an image processing method, and aprogram capable of suppressing unnaturalness of color of an objectextracted from a color image.

The following invention aspects are provided in order to achieve theobject.

An image processing device according to a first aspect is an imageprocessing device comprising: an object area determination unit thatdetermines an object area of a color image; a mask image generation unitthat generates a mask image of the object area; an object imagegeneration unit that extracts the object area and generates an objectimage on the basis of the object area; a color determination unit thatdetermines a color to be subtracted from colors to be applied to theobject area, and sets a smaller number of colors than those of theobject area as colors of the object image; and a probability calculationunit that calculates a probability of pixels of the mask image beingpixels of the object area, in which the object image generation unitsets the colors set using the color determination unit as colors of anedge area of the object image on the basis of the probability.

According to the first aspect, the color of the edge area of the objectimage corresponding to the boundary area of the mask image is set on thebasis of the probability of being an object area, which has been set inthe boundary area of the mask image. Accordingly, inclusion of colors ofthe background or the like into the object image can be suppressed.

The edge area of the object image includes pixels at an edge of theobject image. The edge area of the object image may have a width of twoor more pixels.

In a second aspect, in the image processing device according to thefirst aspect, the probability calculation unit may be configured tocalculate the probability of a pixel in a boundary area in the maskimage being the object area.

According to the second aspect, it is possible to reduce a processingload of computation, as compared with a case in which the probability ofbeing an object area is applied to all pixels of the mask image.

According to a third aspect, the image processing device according tothe first aspect or the second aspect may be configured to include adegree-of-transparency setting unit that sets a degree of transparencyof the edge area of the object area on the basis of the probability; anda pixel integration unit that integrates pixels to which the degree oftransparency has been applied into the object image.

According to the third aspect, an appropriate degree of transparency isset at the edge of the object image. Accordingly, occurrence of jaggiesor the like at the edge of the object image is suppressed.

In a fourth aspect, in the image processing device according to any oneof the first to third aspects, the color determination unit may beconfigured to set a color to be applied to the object image from colorsto be applied to the object area.

In a fifth aspect, in the image processing device according to any oneof the first to fourth aspects, the color determination unit may beconfigured to set a color to be applied to the object image frompredetermined defined colors.

In a sixth aspect, in the image processing device of the fifth aspect,the color determination unit may be configured to set a color designatedby a user from the defined colors as a color to be applied to the objectimage.

In the sixth aspect, a setting screen for setting a defined color may bedisplayed on a display device of a user terminal, and the user maydesignate the color to be applied to the object image using the settingscreen.

In a seventh aspect, in the image processing device according to any oneof the first to sixth aspects, the color determination unit may beconfigured to determine a color to be subtracted from the colors appliedto the object area, using machine learning.

In an eighth aspect, is the image processing device according to any oneof the first to seventh aspects, in a case where pixels of differentcolors are adjacent to each other in the object area, the object imagegeneration unit may be configured to set an intermediate color betweenthe different colors in the adjacent pixels of different colors.

According to the eighth aspect, it is possible to suppress unnaturalnessbetween colors in the object image.

In a ninth aspect, in the image processing device according to any oneof the first to the eighth aspects, the object area determination unitmay be configured to determine the object area on the basis of an areadesignated by a user.

In a tenth aspect, in the image processing device according to any oneof the first to eighth aspects, the object area determination unit maybe configured to determine the object area using machine learning.

In an eleventh aspect, in the image processing device according to anyone of the first to tenth aspects, the object area may be configured toinclude a face area.

In a twelfth aspect, the image processing device according to any one ofthe first to eleventh aspects may be configured to include a stampgeneration unit that generates a stamp on the basis of the object image.

An image processing system according to a thirteenth aspect of thepresent invention is an image processing system comprising a serverdevice connected to a network, the server device including: an objectarea determination unit that determines an object area of a color image;a mask image generation unit that generates a mask image of the objectarea; an object image generation unit that extracts the object area andgenerates an object image on the basis of the object area; a colordetermination unit that determines a color to be subtracted from colorsto be applied to the object area, and sets a smaller number of colorsthan those of the object area as colors of the object image; and aprobability calculation unit that calculates a probability of pixels ofthe mask image being pixels of the object area, in which the objectimage generation unit sets the colors designated using the colordetermination unit as colors of an edge area of the object image on thebasis of the probability.

According to the thirteenth aspect, it is possible to obtain the sameeffects as those in the first aspect.

In the thirteenth aspect, the same matters as the matters specified inthe second to twelfth aspects can be combined appropriately. In thiscase, the component serving the process or function specified in theimage processing device can be ascertained as a component of the imageprocessing system serving the process or function corresponding thereto.

An image processing method according to a fourteenth aspect is an imageprocessing method including: an object area determination step ofdetermining an object area of a color image; a mask image generationstep of generating a mask image of the object area; an object imagegeneration step of extracting the object area and generating an objectimage on the basis of the object area; a color determination step ofdetermining a color to be subtracted from colors to be applied to theobject area, and setting a smaller number of colors than those of theobject area as colors of the object image; and a probability calculationstep of calculating a probability of pixels of the mask image beingpixels of the object area, in which the object image generation stepsets the colors set in the color determination step as colors of an edgearea of the object image on the basis of the probability.

According to the fourteenth aspect, it is possible to obtain the sameeffects as those of the first aspect.

In the fourteenth aspect, the same matters as the matters specified inthe second to twelfth aspects can be combined appropriately. In thiscase, the component serving the process or function specified in theimage processing device can be ascertained as a component of the imageprocessing system serving the process or function corresponding thereto.

A program according to a fifteenth aspect is a program causing acomputer to realize: an object area determination function ofdetermining an object area of a color image; a mask image generationfunction of generating a mask image of the object area; an object imagegeneration function of extracting the object area and generating anobject image on the basis of the object area; a color determinationfunction of determining a color to be subtracted from colors to beapplied to the object area, and setting a smaller number of colors thanthose of the object area as colors of the object image; and aprobability calculation function of calculating a probability of pixelsof the mask image being pixels of the object area, in which the objectimage generation function includes setting the colors set using thecolor determination function as colors of an edge area of the objectimage on the basis of the probability.

According to the fifteenth aspect, it is possible to obtain the sameeffects as those of the first aspect.

In the fifteenth aspect, the same matters as the matters specified inthe second to twelfth aspects can be combined appropriately. In thiscase, the component serving the process or function specified in theimage processing device can be ascertained as a component of the imageprocessing system serving the process or function corresponding thereto.

According to the present invention, the color of the edge area of theobject image corresponding to the boundary area of the mask image is seton the basis of the probability of being an object area, which has beenset in the boundary area of the mask image. Accordingly, inclusion ofcolors of the background or the like into the object image can besuppressed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an image processing device.

FIG. 2 is a diagram illustrating an example of an input image.

FIG. 3 is a diagram illustrating an example of a mask image.

FIG. 4 is a partially enlarged diagram of a boundary between a mask areaand a non-mask area in the mask image.

FIG. 5 is a diagram illustrating an example of pixels in a boundary areain which probability of being an object area has been set.

FIG. 6 is a diagram illustrating an example of an object image.

FIG. 7 is a block diagram illustrating a hardware configuration of theimage processing device.

FIG. 8 is a flowchart illustrating a procedure of an image processingmethod.

FIG. 9 is a block diagram of an image processing system according to anexample of application to a network system.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will bedescribed in detail with reference to the attached drawings. In thepresent specification, the same components are denoted by the samereference numerals, and repeated description will be appropriatelyomitted.

Image Processing Device Description of Function of Image ProcessingDevice

FIG. 1 is a functional block diagram of an image processing device.

The image processing device 10 includes an image acquisition unit 12, anobject area determination unit 14, a mask image generation unit 16, aprobability calculation unit 18, a color determination unit 20, anobject image generation unit 24, and a storage unit 26. The imageprocessing device 10 further includes a degree-of-transparency settingunit 28, a pixel integration unit 30, and a stamp generation unit 32.

The image processing device 10 automatically determines an object areafrom a color image, extracts the object area, and generates an objectimage in which the number of colors is reduced as compared with theobject. The image processing device 10 generates an illustrative stampusing the object image in which the number of colors has been reduced ascompared with the object of the color image.

The image acquisition unit 12 acquires the color image. The imageacquisition unit 12 stores the acquired image in the storage unit 26.The storage unit 26 illustrated in FIG. 1 is a generic term for astorage unit that stores various pieces of data and the like. Forexample, the storage unit 26 is configured using a plurality of storagedevices for each type of data.

An example of the color image is a full color image in which each colorof red, green and blue is represented by 8 bits and color information isrepresented by a value of 24 bits. In a case where a face of a person isextracted from a color image of the person and a stamp of the face ofthe person is generated, the image acquisition unit 12 acquires a colorimage in which the face of the person is captured near a center of theimage.

In a case where a color image in which the face of the person is notcaptured near the center of the image is acquired, a position of theobject area including the face of the person, and a size of the objectarea are determined from the color image. The face of the personillustrated in the embodiment is an example of a face area.

In the present specification, the term image may be used with a meaningof image data. That is, the image acquisition unit 12 acquires imagedata obtained by imaging using an imaging device.

FIG. 2 is a diagram illustrating an example of an input image. An inputimage 100 illustrated in FIG. 2 has a signboard 102 captured as a mainsubject. Further, in the input image 100, a partition 104 is captured asa background. In the signboard 102, two colors different from a ground108 are used for a letter 106. For example, green is used for the ground108. White is used for the letter 106A. Red is used for the letter 106B.

The object area determination unit 14 automatically determines thesignboard 102 illustrated in FIG. 2 as an object area of the input image100. The object area determination unit 14 stores information on theobject area in the storage unit 26.

For an automatic determination of the object area, schemes such asmachine learning such as deep learning, and graph cutting can be used.An example of the deep learning is a convolutional neural network. Theconvolutional neural network may be referred to as a CNN using anabbreviation of the convolutional neural network in English.

The object area determination unit 14 to which the convolutional neuralnetwork is applied includes an input layer, an intermediate layer, andan output layer. The intermediate layer includes a plurality of setsconsisting of a convolutional layer and a pooling layer, and a fullycoupled layer. Each layer has a structure in which a plurality of nodesare connected using an edge.

The input image 100 is input to the input layer. The intermediate layerextracts a feature from the input image 100 input from the input layer.The convolutional layer performs filter processing on a nearby nodeclose to a previous layer to acquire a feature map. The convolutionlayer performs convolution computation using a filter as filterprocessing.

In the pooling layer, the feature map output from the convolution layeris reduced and a new feature map is obtained. The convolutional layerserves as feature extraction such as edge extraction from the imagedata. The pooling layer serves to provide robustness so that theextracted feature is not affected by translation or the like.

For the intermediate layer, an aspect in which the convolutional layeris continuous and an aspect in which a normalization layer is includedmay be adopted. Further, a weight and a bias of a filter to be used ineach convolutional layer are automatically learned using a large numberof pieces of learning data in advance.

In a case where the user designates an object in the color image, theobject area determination unit 14 may determine an object area includingan object designated by the user in the color image.

For example, in a case where the face of the person is not located at acenter of the color image, the user may specify the object in the colorimage by tapping a position of the face of the color image displayed onthe display of the smartphone.

The mask image generation unit 16 illustrated in FIG. 1 generates a maskimage. The mask image generation unit 16 stores the mask image in thestorage unit 26. FIG. 3 is a diagram illustrating an example of the maskimage. The mask image 110 illustrated in FIG. 3 includes a mask area112, a non-mask area 114, and a boundary area 116.

The mask area 112 is an area corresponding to an area other than theobject area of the color image. The non-mask area 114 is an areacorresponding to the object area of the color image. The boundary area116 is an area including a boundary between the mask area 112 and thenon-mask area 114. The boundary area 116 may have a width of one pixelor may have a width of two or more pixels.

FIG. 4 is a partially enlarged diagram of a boundary between the maskarea and the non-mask area in the mask image. FIG. 4 is an enlargeddiagram of a portion denoted by reference numeral 116A in FIG. 3.Further, a black square illustrated in FIG. 4 is a pixel of the maskarea 112. Further, a white square is a pixel of the non-mask area 114.

Reference numeral 117 indicates the boundary between the mask area 112and the non-mask area 114. The boundary area 116 illustrated in FIG. 5has a width of one pixel from the boundary 117 to the mask area 112 anda width of one pixel from the boundary 117 to the non-mask area 114.

The probability calculation unit 18 calculates a probability of eachpixel in the boundary area 116 of the mask image 110 being an objectarea. The probability calculation unit 18 stores the probability ofbeing the object area, which is set in the pixel in the boundary area116 of the mask image 110, in the storage unit 26.

That is, a probability of being an object area, which is representedusing a numerical value exceeding 0% and less than 100%, is set in thepixel in the boundary area 116 of the mask image 110.

The probability calculation unit 18 may calculate, as the probability ofbeing an object area, a degree of reliability included in the objectarea for the pixels on a closed curve representing the boundary of theobject area. The closed curve representing the boundary of the objectarea may have a width of one pixel or may have a width of two or morepixels.

A reciprocal of a squared error of a pixel value in a specific pixel ona closed curve representing an edge of the object area with respect toan average of the pixel values of the pixels in the object area iscalculated. This value represents a degree of reliability of thespecific pixel on the closed curve representing the edge of the objectarea being included in the object area.

In a case where the number of pixels on the closed curve is relativelyincreased, the accuracy of the probability of being an object area isimproved. On the other hand, in a case where the number of pixels on theclosed curve is relatively decreased, the amount of calculation of theprobability of being an object area is relatively decreased.

In a case where the closed curve has any number of pixels from twopixels to ten pixels, a balance between the accuracy of the probabilityof being an object area and the amount of calculation of the probabilityof being an object area is good. The calculation of the probability ofbeing an object area is not limited to the scheme, and another schememay be applied.

FIG. 5 is a diagram illustrating an example of pixels in the boundaryarea in which the probability of being an object area has been set. In apixel 116B, the probability of being an object area is 90%. A numericalvalue assigned to each of a pixel 116C, a pixel 116D, and a pixel 116Frepresents the probability of being an object area of each pixel.

The pixel 112A is a pixel in the mask area 112. In the pixel in the maskarea 112, probability of being an object area is 0%. Further, a pixel114A is a pixel in the non-mask area 114. In the pixel 114A, theprobability of being an object area is 100%.

The probability calculation unit 18 may calculate the probability ofbeing an object area for all pixels of the mask image. The mask imagegeneration unit 16 may set a pixel with the probability of 0% as thepixel in the mask area 112, set a pixel with the probability of 100% asa pixel in the non-mask area, and sets pixel with the probabilityexceeding 0% and less than 100% as a pixel in the boundary area.

The color determination unit 20 determines a color to be subtracted innumber from colors applied to the object area. As an example of colorreduction may include an example in which 1677 million colors, which are24-bit full colors, are reduced to any number of colors from threecolors to tens of colors.

For a color reduction number, a numerical value indicating validity canbe calculated, and the color reduction number can be defined from thenumber of colors that do not cause a sense of color discomfort in animage in which the number of colors is reduced. For example, energy maybe calculated as the numerical value indicating validity, and the colorreduction number may be defined from the number of colors in which theenergy is stable.

In a case where the stamp is generated using the object image, the stamphas an illustrative element. In the illustrative stamp, it is notnecessary to apply full color and perform color representation, but itis possible to apply the number of colors in which certain quality ofthe stamp is maintained.

For example, in a color image that is an original image of the stamp, asimilar color is represented in multi-gradation. On the other hand, itis convenient to use the similar color as a stamp in a case where thesimilar color is represented by one or several colors.

The color determination unit 20 may set a color to be applied to theobject image from among the colors to be applied to the object area. Theobject image is generated by extracting the object area from the colorimage.

The color determination unit 20 may automatically determine the color tobe subtracted from the object area. For example, the intermediate colormay be determined as the color to be subtracted. The color determinationunit 20 may determine a color designated in a color palette representingthe color to be used for the object area to be a color to be applied tothe object image. The color designated in the color palette illustratedin the embodiment is an example of the defined color.

The object image generation unit 24 generates an object image using thecolor image and the mask image. The object image generation unit 24applies the color designated using the color determination unit 20 toeach pixel of the object image. That is, the object image generationunit 24 reduces the color of the object area to generate an objectimage.

FIG. 6 is a diagram illustrating an example of the object image. Anobject image 120 illustrated in FIG. 6 is reduced in color as comparedwith the signboard 102 illustrated in FIG. 2. For example, for a letter122A of the object image 120 corresponding to the letter 106A of theinput image 100, black is applied to the letter 106A to whichmulti-gradation gray has been applied.

Similarly, for a letter 122B of the object image 120 corresponding tothe letter 106B of the input image 100, one type of red is applied to aletter 106B to which multi-gradation red has been applied. Further, fora ground 128 of the object image 120 corresponding to the ground 108 ofthe input image 100, one type of green is applied to the ground 108 towhich multi-gradation green has been applied.

Further, the pixel of the edge 120A of the object image 120 isdetermined in color on the basis of the probability of being an objectarea, which has been set in the boundary area 116 of the mask image 110.For example, in a case where the determination threshold is 50%, thecolor of the object image is applied to the pixel of which a probabilityof being an object area is 50% or more.

Accordingly, a setting of colors other than the object area as the colorof the object image can be suppressed, and unnaturalness of the color ofthe object image can be suppressed. 50% of the determination thresholdis an example, and any determination threshold may be defined accordingto generation conditions of the object image such as image quality ofthe object image and the color to be subtracted.

The degree-of-transparency setting unit 28 sets a degree of transparencyin the pixels in the edge area of the object image 120 according to theprobability of being an object area, which is applied to the boundaryarea 116 of the mask image 110. The edge area is an area having a widthof at least one pixel including the edge 120A of the object image 120.

For example, in a pixel having a relatively high probability of being anobject area, the degree of transparency is made relatively low such thata degree at which the background is transmitted onto the object area isdecreased. On the other hand, in a pixel having a relatively lowprobability of being an object area, the degree of transparency is maderelatively high such that the degree at which the background istransmitted onto the object area is increased. That is, the degree oftransparency is represented as the degree at which the background istransmitted onto the object area. For example, in a case where thedegree of transparency is 0%, the background is not transmitted onto theobject area at all. On the other hand, in a case where the degree oftransparency is 100%, the background is completely transmitted to theobject area.

The pixel integration unit 30 integrates the pixels of which the degreeof transparency has been set using the degree-of-transparency settingunit 28 into the object image 120. The pixel integration unit 30 stores,in the storage unit 26, the object image into which the pixels of whichthe degree of transparency has been set are integrated.

The pixels of which the degree of transparency has been set areintegrated near the edge 120A and the edge 120A of the object image 120.Accordingly, generation of jaggies at the edge 120A of the object image120 can be suppressed by blurring the vicinity of the edge 120A and theedge 120A of the object image 120.

The stamp generation unit 32 generates a stamp from the object imagegenerated using the object image generation unit 24. The stampgeneration unit 32 stores the generated stamp in the storage unit 26.

Since the stamp is reduced in color as compared with the object area ofthe color image, the illustrative stamp can be generated. Further, inthe object image 120 that is a basis of the stamp, color unnaturalnessat the edge 120A is suppressed, and the occurrence of jaggies issuppressed. Accordingly, a stamp in which the color unnaturalness hasbeen suppressed and the occurrence of the jaggies has been suppressedcan be generated.

Description of Hardware Configuration of Image Processing Device

FIG. 7 is a block diagram illustrating a hardware configuration of theimage processing device.

Overall Configuration

The image processing device 10 illustrated in FIG. 1 includes a controlunit 40, a memory 42, a storage device 44, a network controller 46, anda power supply device 48. The control unit 40, the memory 42, thestorage device 44, and the network controller 46 are communicativelyconnected via the bus 41.

The image processing device 10 may include a display controller 52, aninput and output interface 54, and an input controller 56. The imageprocessing device 10 can execute a defined program using the controlunit 40 to realize various functions of the image processing device 10.

Control Unit

The control unit 40 functions as an entire control unit, variouscomputation units, and a storage control unit of the image processingdevice 10. The control unit 40 executes a program stored in a read onlymemory (ROM) included in the memory 42.

The control unit 40 may download a program from an external storagedevice via the network controller 46 and execute the downloaded program.The external storage device may be communicatively connected to theimage processing device 10 via the network 50.

The control unit 40 uses a random access memory (RAM) included in thememory 42 as a computation area, and executes various processes incooperation with various programs. Accordingly, various functions of theimage processing device 10 are realized.

The control unit 40 controls reading of data from the storage device 44and writing of data to the storage device 44. The control unit 40 mayacquire various pieces of data from the external storage device via thenetwork controller 46. The control unit 40 can execute various processessuch as computation using the acquired various pieces of data.

The control unit 40 may include one or more processors. Examples of theprocessor include a field programmable gate array (FPGA) and aprogrammable logic device (PLD). The FPGA and the PLD are devices ofwhich a circuit configuration can be changed after manufacture.

Another example of the processor may include an application specificintegrated circuit (ASIC). The ASIC includes a circuit configurationthat is specifically designed to perform a specific process.

For the control unit 40, two or more processors of the same type can beapplied. For example, for the control unit 40, two or more FPGAs may beused or two PLDs may be used. For the control unit 40, two or moreprocessors of different types can be applied. For example, for thecontrol unit 40, one or more FPGAs and one or more ASICs can be applied.

In a case where a plurality of control units 40 are included, theplurality of control units 40 may be configured using one processor. Anexample in which the plurality of control units 40 are configured usingone processor includes an aspect in which one processor is configuredusing a combination of one or more central processing units (CPUs) andsoftware, and this processor functions as the plurality of control units40. The software in the present specification is synonymous with aprogram.

A graphics processing unit (GPU) that is a processor specialized forimage processing may be applied instead of the CPU or in combinationwith the CPU. A representative example in which the plurality of controlunits 40 are configured using one processor may include a computer.

Another example in which the plurality of control units 40 areconfigured using one processor may include an aspect in which aprocessor that realizes functions of the entire system including theplurality of control units 40 using one IC chip is used. Arepresentative example of a processor that realizes the functions of theentire system including the plurality of control units 40 using one ICchip may include a system on chip (SoC). The IC is an abbreviation ofintegrated circuit.

Thus, the control unit 40 is configured using one or more of variousprocessors as a hardware structure.

Memory

The memory 42 includes a ROM (not illustrated) and a RAM (notillustrated). The ROM stores various programs that are executed by theimage processing device 10. The ROM stores parameters, files, and thelike that is used for execution of various programs. The RAM functionsas a temporary storage area for data, a work area for the control unit40, and the like.

Storage Device

The storage device 44 stores various pieces of data non-temporarily. Thestorage device 44 may be externally attached to the image processingdevice 10. A large capacity semiconductor memory device may be appliedinstead of or in combination with the storage device 44.

Network Controller

The network controller 46 controls data communication with an externaldevice. Control of the data communication may include management of datacommunication traffic. For the network 50 connected via the networkcontroller 46, a known network such as a local area network (LAN) can beapplied.

Power Supply Device

As the power supply device 48, a large capacity power supply device suchas an uninterruptible power supply (UPS) is applied. The power supplydevice 48 supplies power to the image processing device 10 in a casewhere a commercial power is shut off due to a power failure or the like.

Display Controller

The display controller 52 functions as a display driver that controlsthe display unit 60 on the basis of a command signal transmitted fromthe control unit 40.

Input and output Interface

The input and output interface 54 communicatively connects the imageprocessing device 10 to an external device. The input and outputinterface 54 can apply a communication standard such as a universalserial bus (USB).

Input Controller

The input controller 56 converts a format of the signal input using themanipulation unit 62 into a format suitable for the process of the imageprocessing device 10. Information input from the manipulation unit 62via the input controller 56 is transmitted to each unit via the controlunit 40.

A hardware configuration of the image processing device 10 illustratedin FIG. 7 is an example, and addition, deletion, and change can be madeappropriately.

Flowchart of Image Processing Method

FIG. 8 is a flowchart illustrating a procedure of the image processingmethod. The image processing method of which the procedure isillustrated in FIG. 8 includes an input image acquisition step S10, anobject area determination step S12, a mask image generation step S14, acolor determination step S16, a probability calculation step S18, anobject image generation step S20, and a stamp generation step S22.

In the input image acquisition step S10, the image acquisition unit 12illustrated in FIG. 1 acquires the input image 100 illustrated in FIG. 2as a color image. After the input image acquisition step S10, an inputimage storage step of storing the input image 100 may be performed.After the input image acquisition step S10, the process proceeds to theobject area determination step S12.

In the object area determination step S12, the object area determinationunit 14 automatically determines an object area from the input image100. After the object area determination step S12, an object informationstorage step of storing information on the object area may be performed.After the object area determination step S12, the process proceeds tothe mask image generation step S14.

In the mask image generation step S14, the mask image generation unit 16generates the mask image 110 illustrated in FIG. 3. After the mask imagegeneration step S14, a mask image storage step of storing the mask image110 may be performed. After the mask image generation step S14, theprocess proceeds to the color determination step S16.

In the color determination step S16, the color determination unit 20determines the color to be subtracted, from the colors applied to thesignboard 102 illustrated in FIG. 2, which is an object. For example, anintermediate color to be applied to the ground of the signboard 102 andan intermediate color to be applied to a letter may be determined to bethe colors to be subtracted. After the color determination step S16, theprocess proceeds to the probability calculation step S18.

In the probability calculation step S18, the probability calculationunit 18 calculates the probability of the pixel in the boundary area 116of the mask image 110 being an object area, as illustrated in FIG. 5.After the probability calculation step S18, the process proceeds to theobject image generation step S20.

In the object image generation step S20, the object image generationunit 24 generates the object image 120 illustrated in FIG. 6. The objectimage generation step S20 includes a color reduction processing step ofreducing the color to be applied to the object area on the basis of thecolor to be subtracted, which has been determined in the colordetermination step S16.

The object image generation step S20 may include adegree-of-transparency setting step of setting a degree of transparencyin the pixels in the edge area of the object image according to theprobability of being an object area, which has been set in the pixel inthe boundary area 116 of the mask image 110.

Further, the object image generation step S20 may include a pixelintegration step of integrating pixels, of which the degree oftransparency has been set in the degree-of-transparency setting step,into an object image. After the object image generation step S20, theprocess proceeds to the stamp generation step S22.

In the stamp generation step S22, the stamp generation unit 32 generatesa stamp on the basis of the object image. After the stamp generationstep S22, a stamp storage step of storing a stamp may be performed.Further, after the stamp generation step S22, a stamp output step ofoutputting a stamp may be performed. An example of the stamp output mayinclude a display of a stamp using a display of a smartphone that isused by the user.

Although the image processing method including the stamp generation stepS22 has been described in the embodiment, the stamp generation step S22may be changed to a generation step other than the stamp generationusing the object image. For example, an ornament generation step ofgenerating an ornament using the object image may be applied instead ofthe stamp generation step S22.

Operation and Effects

With the image processing device and the image processing methodconfigured as described above, it is possible to obtain the followingoperation and effects.

[1] The colors of the object image generated using the object areaextracted from the color image are reduced in number as compared withthe colors to be applied to the object area. The pixels at the edge ofthe object image have colors to be applied to the object image, on thebasis of the probability of being an object area in the color image.Accordingly, inclusion of colors other than the colors to be applied tothe object image into the object image is suppressed, and unnaturalnessof the colors in the object image can be suppressed.

[2] In the edge area of the object image, the degree of transparency seton the basis of the probability of being an object area in the colorimage is integrated. Accordingly, the edge area of the object image hasan appropriate degree of transparency, and an object image in whichjaggies or the like has been suppressed can be generated.

[3] An object area is automatically extracted from the color image. Forthe automatic extraction of the object area, deep learning is applied.Accordingly, it is possible to increase the accuracy of object areaextraction.

Application Example

An intermediate color between respective colors may be applied to thepixel at the color boundary in the object image 120 illustrated in FIG.6 using a probability of being each color. For example, in the objectimage 120, a probability of being a letter is calculated for the pixelsin the edge area of the letter that is the boundary area between thecolor of the letter and the color of the background. To calculate theprobability of being a letter, a scheme of calculating the probabilityof being an object area in the color image can be applied. An errordiffusion method may be applied in a case where colors to be applied tothe object area are reduced in number.

The intermediate color may be a single gradation or multi-gradation. Awidth of the boundary area of the color may be one pixel or may be twoor more pixels. The pixels in the edge area of the letter illustrated inthe embodiment are examples of pixels with adjacent different color.

Operation and Effects of Application Example

According to the image processing device and the image processing methodof the application example, an intermediate color based on a probabilityof being one color is applied to the pixels in the boundary area of thecolors in the object image. Accordingly, it is possible to suppress theoccurrence of unnaturalness between the colors of the object image 120.

Application to Network System

For the image processing device 10 illustrated in FIG. 1, a desktop typecomputer can be applied. For the image processing device 10, a portableinformation processing terminal such as a smartphone, a tablet computer,and a notebook computer may be applied.

FIG. 9 is a block diagram of an image processing system according to anexample of application to a network system. An image processing system400 illustrated in FIG. 9 includes a server device 410. The serverdevice 410, the first user terminal 420, the second user terminal 422,and the third user terminal 424 are communicatively connected via thenetwork 402. The image processing system 400 may include a mass storagedevice such as a storage device communicatively connected via thenetwork 402.

For the network 402, a wide area communication network such as wide areanetwork (WAN) may be applied, or a local area communication network suchas local area network (LAN) may be applied. A communication scheme,communication protocol, and the like of the network 402 are not limited.For the network 50 illustrated in FIG. 7, the network 402 illustrated inFIG. 9 can be applied.

For the server device 410, the image processing device 10 described withreference to FIGS. 1 to 8 is applied. In the aspect illustrated in FIG.9, the display controller 52, the input and output interface 54, theinput controller 56, the display unit 60, and the manipulation unit 62illustrated in FIG. 2 may be omitted.

In the server device 410 illustrated in FIG. 9, the storage device 44illustrated in FIG. 2 may be communicatively connected to the serverdevice 410 via the network 402. FIG. 9 illustrates an example in which aportable terminal is applied as the first user terminal 420 and thesecond user terminal 422, and a laptop computer is applied as the thirduser terminal 424. A user terminal such as the first user terminal 420may be a device that is communicatively connected to the server device410 via the network 402.

Example of Application to Program

The image processing device 10 and the image processing method describedabove can be configured as a program causing a function corresponding toeach unit in the image processing device 10 or a function correspondingto each step in the image processing method to be executed using acomputer.

Examples of functions corresponding to the respective steps include aninput image acquisition function, an object area determination function,a mask image generation function, a color determination function, aprobability calculation function, an object image generation function, adegree-of-transparency setting function, and a pixel integrationfunction.

The input image acquisition function corresponds to the imageacquisition unit 12 illustrated in FIG. 1. The object area determinationfunction corresponds to the object area determination unit 14. The maskimage generation function corresponds to the mask image generation unit16.

The probability calculation function corresponds to the probabilitycalculation unit 18. The color determination step corresponds to thecolor determination unit 20. The object image generation functioncorresponds to the object image generation unit 24. The storage stepcorresponds to the storage unit 26. The degree-of-transparency settingfunction corresponds to the degree-of-transparency setting unit 28. Thepixel integration function corresponds to the stamp generation unit 30.The stamp generation function corresponds to the stamp generation unit32.

It is possible to store the program causing the computer to realize theinformation processing function described above in a computer-readableinformation storage medium, which is a tangible non-temporaryinformation storage medium, and provide the program through theinformation storage medium. Further, an aspect in which a program signalis provided via a network is also possible, instead of the aspect inwhich the program is stored in the non-temporary information storagemedium and provided.

Combination of Embodiment and Modification Example

The components described in the above-described embodiment and thecomponents described in the application example or the like can be usedin appropriate combination, and some of the components can be replaced.

In the embodiment of the present invention described above, it ispossible to appropriately change, add, or delete constituentrequirements without departing from the spirit of the present invention.The present invention is not limited to the embodiments described above,and many modifications can be made by those skilled in the art withinthe technical spirit of the present invention.

EXPLANATION OF REFERENCES

-   10: image processing device-   12: image acquisition unit-   14: object area determination unit-   16: mask image generation unit-   18: probability calculation unit-   20: color determination unit-   24: object image generation unit-   26: storage unit-   28: degree-of-transparency setting unit-   30: pixel integration unit-   32: stamp generation unit-   40: control unit-   41: bus-   42: memory-   44: storage device-   46: network controller-   48: power supply device-   50: network-   52 display controller-   54: input and output interface-   56: input controller-   60: display unit-   62: manipulation unit-   100: input image-   102: signboard-   104: partition-   106: letter-   106A: letter-   106B: letter-   108: ground-   110: mask image-   112: mask area-   114: non-mask area-   114A: pixel-   116: boundary area-   116A: pixel-   116B: pixel-   116C: pixel-   116D: pixel-   116E: pixel-   116F: pixel-   117: boundary-   120: object image-   120A: edge-   122A: letter-   122B: letter-   128: ground-   400: image processing system-   402: network-   410: server device-   420: first user terminal-   422: second user terminal-   424: third user terminal-   S10 to S22: Respective steps of image processing method

What is claimed is:
 1. An image processing device comprising: an objectarea determination unit that determines an object area of a color image;a mask image generation unit that generates a mask image of the objectarea; an object image generation unit that extracts the object area andgenerates an object image on the basis of the object area; a colordetermination unit that determines a color to be subtracted from colorsto be applied to the object area, and sets a smaller number of colorsthan those of the object area as colors of the object image; and aprobability calculation unit that calculates a probability of pixels ofthe mask image being pixels of the object area, wherein the object imagegeneration unit sets the colors set using the color determination unitas colors of an edge area of the object image on the basis of theprobability.
 2. The image processing device according to claim 1,wherein the probability calculation unit calculates the probability of apixel in a boundary area in the mask image being the object area.
 3. Theimage processing device according to claim 1, comprising: adegree-of-transparency setting unit that sets a degree of transparencyof the edge area of the object area on the basis of the probability; anda pixel integration unit that integrates pixels to which the degree oftransparency has been applied into the object image.
 4. The imageprocessing device according to claim 1, wherein the color determinationunit sets a color to be applied to the object image from colors to beapplied to the object area.
 5. The image processing device according toclaim 1, wherein the color determination unit sets a color to be appliedto the object image from predetermined defined colors.
 6. The imageprocessing device according to claim 5, wherein the color determinationunit sets a color designated by a user from the defined colors as thecolor to be applied to the object image.
 7. The image processing deviceaccording to claim 1, wherein the color determination unit determinesthe color to be subtracted from the colors applied to the object area,using machine learning.
 8. The image processing device according toclaim 1, wherein, in a case where pixels of different colors areadjacent to each other in the object area, the object image generationunit sets an intermediate color between the different colors in theadjacent pixels of different colors.
 9. The image processing deviceaccording to claim 1, wherein the object area determination unitdetermines the object area on the basis of an area designated by a user.10. The image processing device according to claim 1, wherein the objectarea determination unit determines the object area using machinelearning.
 11. The image processing device according to claim 1, whereinthe object area includes a face area.
 12. The image processing deviceaccording to claim 1, further comprising: a stamp generation unit thatgenerates a stamp on the basis of the object image.
 13. An imageprocessing system comprising a server device connected to a network, theserver device comprising: an object area determination unit thatdetermines an object area of a color image; a mask image generation unitthat generates a mask image of the object area; an object imagegeneration unit that extracts the object area and generates an objectimage on the basis of the object area; a color determination unit thatdetermines a color to be subtracted from colors to be applied to theobject area, and sets a smaller number of colors than those of theobject area as colors of the object image; and a probability calculationunit that calculates a probability of pixels of the mask image beingpixels of the object area, wherein the object image generation unit setsthe colors set using the color determination unit as colors of an edgearea of the object image on the basis of the probability.
 14. An imageprocessing method comprising: an object area determination step ofdetermining an object area of a color image; a mask image generationstep of generating a mask image of the object area; an object imagegeneration step of extracting the object area and generating an objectimage on the basis of the object area; a color determination step ofdetermining a color to be subtracted from colors to be applied to theobject area, and setting a smaller number of colors than those of theobject area as colors of the object image; and a probability calculationstep of calculating a probability of pixels of the mask image beingpixels of the object area, wherein the object image generation step setsthe colors set in the color determination step as colors of an edge areaof the object image on the basis of the probability.
 15. Anon-transitory, computer-readable tangible recording medium whichrecords a program causing a computer to realize: an object areadetermination function of determining an object area of a color image; amask image generation function of generating a mask image of the objectarea; an object image generation function of extracting the object areaand generating an object image on the basis of the object area; a colordetermination function of determining a color to be subtracted fromcolors to be applied to the object area, and setting a smaller number ofcolors than those of the object area as colors of the object image; anda probability calculation function of calculating a probability ofpixels of the mask image being pixels of the object area, wherein theobject image generation function includes setting the colors set usingthe color determination function as colors of an edge area of the objectimage on the basis of the probability.